Bottom-up Attention Improves Action Recognition Using Histograms of Oriented Gradients

When recognizing others’ action, we pay attention to their body parts and/or objects they are manipulating rather than observing their whole body movement. Bottom-up saliency is a promising cue to determine where to attend and hence to identify what the persons are doing because their body parts acting on objects become more conspicuous when contributing to the action. This paper proposes an architecture for action recognition that integrates bottom-up saliency with Histograms of Oriented Gradients (HOG). The HOG extracts the local features of others’ action while the saliency gives an attentional weight to the HOG descriptor. Our experiments demonstrate that the saliency-based attention improves the performance of action recognition by emphasizing the HOG features relevant to the action.